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FIELD AND BACKGROUND OF THE INVENTION 

The underlying invention generally relates to the field of markup languages used to de- 
10 scribe adaptive mobile multimedia applications and/or presentations being inherently de- 
pendent on the continuously changing environment they are running in, which means that 
these mobile multimedia applications and/or presentations need to be adapted to the prefer- 
ences of mobile users, the capabilities of their mobile computing devices, and the users 
physical impact like current location, time or temperature. 

15 

Applications running on mobile devices are experiencing a highly dynamic environment 
They have to cope with inherent fluctuation of the available Quality of Service (QoS) in 
wireless networks, and they should be usable on a wide range of different devices. Addi- 
tionally, for providing a good service, they should take the users 1 preferences and context 

20 into account, e.g. his current location. To meet those requirements, the developers of such 
applications need to be supported by a platform that allows applications to adapt to differ- 
ent networks, device and user situations, and which hides some of the complexity involved. 
Adaptive Multimedia Support Environment (AMUSE) is a platform, which supports adap- 
tive distributed multimedia retrieval applications. The platform consists of three parts, a 

25 client-side middleware, a multimedia application built on top of the middleware, and a 
server-side context-aware mobile portal. 

By contrast, the focus of the underlying invention is the document-based approach in the 
way that adaptive mobile applications and/or presentations are described with an XML- 
30 based document, generated by a server-side authoring system (represented by the context- 
aware mobile portal), processed and rendered by client-side middleware, capable of exe- 
cuting the processing of the XML-based document description. 
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There are different ways to realize multimedia applications. First, a multimedia application 
can be quite specific, implemented for a specifically defined scenario, and the application 
itself might be adjustable by a configuration file, which is quite good for a specific, nar- 

5 rowed scenario. The second way would be to describe a multimedia application in a docu- 
ment-based way and let a browser or player interpret the document, which is done e.g. in 
"Synchronized Multimedia Integration Language" (SMIL, cf. SMIL 2.0, W3C Recommen- 
dation, 7 August, 2001, ht^>://www.w3.org/TR/smil20/), and is a more general approach. 
In the scope of the underlying invention, the latter has been applied. Thereby, multimedia- 

10 document models such as Hypertext Markup Language (HTML), Dynamic HTML 

(DHTML), Multimedia and Hypermedia Information Coding Expert Group (MHEG, cf. 
International Organization for Standardization, ISO/EC JTC1/SC29/WG12, 
http://www.mheg.org/), SMIL and HyTime (cf. Users' Group Home Page www.hytime.org) 
have been evaluated in order to use or extend their document models for describing adap- 

15 tive multimedia applications. Unfortunately, these document models do not provide suffi- 
cient modeling support for adaptive applications. Adaptation is addressed to some extend 
once at start-up time of a multimedia application, but mobile adaptive multimedia applica- 
tions have additional adaptation requirements owing to the inherent dynamic environment 
they are running in. Selecting appropriate streaming media at start-up time does not con- 

20 sider changes, like fluctuation of QoS in wireless network during the run-time of an appli- 
cation. 

Applications running on mobile devices are experiencing a highly dynamic environment. 
The environmental changes can be categorized into three different dimensions: network 

25 QoS, mobile device capabilities, and user context. A mobile application has to cope with 
changes in all three dimensions. Thereby, the term "situation" is used for a given network 
QoS, mobile device capabilities and user context. A situation can be characterized by a set 
of parameters describing different aspects of the environment. The "current situation" is the 
situation, which exists at present for a mobile application. If the current situation causes a 

30 problem for the application or if it allows the application to improve its service, the appli- 
cation may change its behavior to better reflect said current situation. As this is true for all 
mobile applications, the situation is especially interesting when looking at multimedia ap- 
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plications, as those have very tough requirements, especially concerning Quality-of-Service 
(QoS) conditions of the applied network. 

OBJECT OF THE UNDERLYING INVENTION 

5 

In view of the explanations mentioned above, it is the object of the invention to propose a 
technology supporting a personalization and context-aware adaptation of document-based 
mobile multimedia applications. 



10 The multimedia applications can e.g. be retrieved from specific content providers via mo- 
bile networks to the preferences of mobile users, the capabilities of their mobile devices 
and different application scenarios. 

This object is achieved by means of the features of the independent claims. Advantageous 
15 features are defined in the dependent claims. Further objects and advantages of the inven- 
tion are apparent in the following detailed description. 

BRIEF DESCRIPTION OF THE DRAWINGS 

20 Further advantages and possible applications of the underlying invention will become evi- 
dent from the following description of the preferred embodiment of the invention, which is 
depicted in the following drawings: 



Fi & 1 presents a Structure Module according to one embodiment of the underlying 

invention, . . , 
Fig. 2 presents a StructureModule according to an embodiment 

Fig. 3 presents a Medialtems Module according to one embodiment, 

Fig. 4 presents an Interactions Module according to one embodiment, 

Fig. 5 illustrates the relations between the Layout Module, the Medialtems Module, 

an Adaptation Module, and the Interactions Module, 
Fig. 6 exhibits the format of a typical mobile multimedia presentation, 

Fig. 7 shows the description of an AMIL document in the AMIL DOM data structure, 
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Fig. 8 exhibits a Constraint Module according to one embodiment of the underlying 

invention, 

Fig. 9 shows a first table illustrating different adaptation possibilities, 

Fig. 10 shows a second table illustrating the applied association for modeling the link- 

ing between the Adaptation Module and the Medialtems Module, between the 
Medialtems Module and the Layout Module, and between the Interactions 
Module and the Layout Module, 
Fig. 1 1 shows a third table presenting the applied restriction identifier, 

Fig- 12 shows 'a fourth table presenting the applied adaptation, trigger identifier, 

Fig. 13 shows an example for describing different adaptation possibilities (APs), 

Fig. 14 shows an example for calculating priorities, 

Fig. 15 shows an example for adaptation possibilities (APs), 

Fig. 16 shows an example for the linking between the Adaptation Module and the Me- 

dialtems Module, between the Medialtems Module and the Layout Module, 
and between the Interactions Module and the Layout Module, 

Fig. 17 shows an example for the reuse of an AMJL document, 

Fig. 18 shows an example for additional functionality, 

Fig. 19 shows a first example for a document fragment of an adaptation event descrip- 

tion, 

Fig. 20 shows a second example for a document fragment of an adaptation event de- 

scription, 

Fig. 21 shows an example for different adaptation alternatives, 

Fig. 22 shows an example for the definition and usage of constraints, 

Fig. 23 presents a first activity diagram showing a process of determining different 

...... . adaptation possibilities by using the applied AMIL DOM data structure, 

Fig. 24 presents a second activity diagram illustrating changes of the applied AMEL 

DOM data structure, 

Fig. 25 outlines a first UML class diagram showing the relation between media items 

and regions, 

Fig. 26 outlines a second UML class diagram showing the relation between widgets 

and regions, 

Fig. 27 outlines a third UML class diagram illustrating the applied linking elements, 
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Fig. 28 outlines a fourth UML class diagram showing the application programming 

interface (API) of the applied AMIL DOM data structure, 
Fig. 29 gives an overview of fee adaptation description module, 

Fig. 30 exhibits a diagram showing the time-varying quality of the same media item, 

Fig. 3 1 exhibits a first diagram showing the time-varying quality of different media 

items, 

Fig. 32a exhibits a second diagram showing the time-varying quality of different media 
items, 

Fig. 32b exhibits a third diagram showing the time-varying quality of different media 
items, 

Fig. 33 exhibits a fourth diagram showing the time-varying quality of different media 

items, 

Fig. 34 exhibits a fifth diagram showing the time-varying quality of different media 

items, 

Fig. 35 outlines a first sequence diagram showing the usage of the AMIL DOM data 

structure, 

Fig. 36 outlines a second sequence diagram showing possible modifications on the 

AMIL document, 

Fig. 37 illustrates changes of the current situation resulting in modifications of the 

linking structure of the AMDL document, 
Fig. 38 presents a class diagram showing possible events in case the current situation 

has changed, 

Fig. 39 shows an overview diagram of the Events Module supported by AMIL, which 

allows acting on changes of the current situation, and 
Fig. 40 shows an example of an AMIL document supporting the core modules. 

DETAILED DESCRIPTION OF THE UNDERLYING INVENTION 

In the following, the preferred embodiment of the underlying invention as depicted in Figs. 
1 to 40 shall be explained in detail. The meaning of the symbols designated with reference 
signs in Figs. 1 to 40 can be taken from the annexed table. 
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First, the main concepts and characteristics of the new AMIL document model 100 shall be 
introduced. 

As mentioned above, the AMUSE document model is invented due the fact that state-of- 
the-art document models do not, or only to some extend, consider the adaptation aspect in a 
way required if a document-based application runs in a highly dynamic environment. 

The AMIL document model 100 is the link between a technique regarding the authoring 
process of a mobile adaptive multimedia application and an AMEL player, which is doing 
the processing of an AMIL document 2402. The AMIL player can be split up into two 
separate components - a browser and a supporting "client-side middleware". In the scope of 
the underlying invention, these expression are used in the following way: The browser, 
which is also named "Multimedia Browser", executes presentation related functions and 
user interaction 404; whereas the "client-side middleware" provides the functionality for 
executing the adaptation process. 

hi the following section, the AMIL document model 100 according to one embodiment of 
the underlying invention shall be explained. The document is concluded with several de- 
monstrative examples. 

Modules 

The adaptive document model 100 can be split up into different modules according to their 
functionality. This enables the reuse of specific modules as an extension to other document 
models 100. The example 1700 depicted in Fig. 17 explains the reuse of complete AMIL 
documents. Relationships between these modules are realized by linking elements together. 
The AMIL document model 100 supports a "Main Module" 102, which simply serves as a 
container for all others modules. The "Main Module" 102 itself does not provide any fur- 
ther functionality; it serves logically as an overall aggregation/container module. 

Each module is described in the following way: First, a table containing an Unified Mod- 
eling Language (UML) diagram and an Extensible Markup Language (XML) Schema de- 
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scription 700 is shown, and second, the meaning/semantic of each element and attribute is 
explained in detail. In case a clear separation to similar concepts is necessary, the differ- 
ences are set down after the explanation of each element's or attribute's semantic. 

It should be noted that, for describing the syntax, UML and XML Schemata are used. For 
describing XML in UML, UML's extensible mechanism (stereotypes, tagged values arid 
constraints) was applied. Detailed information about this kind of modeling can be found in 
"Modeling XML Applications with UML. Practical e-Business Applications" (ISBN 0- 
201-70915-5) by D. Carlson. 

AMCL (Main) and Core Modules 

The "Core Module" 104 encompasses all module required in each AMTT. document 2402. 
The Extension Module 1 14 can optionally be used for a more comprehensive description of 
the adaptation processes. 

Fig. 1 gives a survey of the modules proposed by the AMEL document model 100. For ' 
modeling, UML "grouping things" are used. Thereby, every module is described by a . 
"package" as described in "The Unified Modeling Language" (User Guide, 
ISBN 0-201-57168-4) by G. Booch, J. Rumbaugh, and I. Jacobson. 

Note: 

Each package logically "depends" (UML relationship "dependency") upon the "Structure 
Module" 1 12. This relationship is not shown in Fig. 1 but still exits. 
The "Core Module" 104 and the "Extension Module" 1 14 are, like the "Main" Module 102, 
simply used for grouping. 

1 . AMEL Structure Module 

The AMEL "Structure Module" 112 reflects the fundamental structure of an AMEL docu- 
ment 2402. Like XHTML, it splits up the document into two sections - a "head module" 
section 202 containing the declaration of the layout 3702, used media items 304, interac- 
tions 404, and e.g. constraints 802 as well as a "body module" section 204 focusing on the 
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adaptation aspect where the intrinsic adaptation possibilities 1500 are described. 
„amil" element: 

The "amil" element is the outermost element that contains all others. 
This element, according to the XML definition, can be preceded and 
followed by comments and processing instructions, but can not be pre- 
ceded by other elements or text nodes, 
"id" attribute: 

Used according to XML as an identifier, for referencing the whole 
AMIL document 2402. 
"version" attribute: 

Used to identify the version of the AMIL document model 100. 

"head module" 202: 

Encompasses all declarations referenced by child elements of the "body 

module" 204. 
"body module" 204: 

Encompasses elements describing the Adaptation Possibilities. 

2. AMEL Layout Module 

The AMIL "Layout Module" 106 is taken on from the SMIL Layout Module as described 
in "Synchronized Multimedia Integration Language" (SMIL 2.0, W3C Recommendation, 
August, 2001, http://www.w3.org/TR/smil20/). Thereby, the elements 



are supported. Detailed information can be found in the SMIL documentation "The SMIL 
2.0 Layout Modules" (http://www.w3.org/TR/smil20/layout.html). 



10 



"smihlayout", 
"smil:region", and 
"smihroot-layout" 



"region" element 502: 
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The element "smihregion" is taken over from the SMIL document model. Its attribute. 
"smihfilT is extended in order to consider the aspect of size adaptation. The state-of-the-art 
usage of this attribute is to specify the behavior if the intrinsic height/width of a media Item 
5 304 is not equal to the dimension of a region 502. 

In AMIL, a region 502 can be used by different media items 304 during playing back a 
multimedia presentation, which means that media items 304 with different formats 600 are 
displayed in a region 502 with a well-defined dimension. Therefore, new attribute values 

10 for the attribute "smil:fill" are specified. The problem is that scaling the visual size of a . 
media requires, under certain circumstance, a lot of CPU power. Especially up-scaling is a 
quite expensive operation. Down-scaling also takes CPU power, but not as much as scaling 
up. A further characteristic in AMIL is that a region 502 can be used by many media items 
304, which can support different formats 600 - e.g. the Common Intermediate Format 

15 (OF), or Quarter GIF (QCIF). At a specific scaling factor, a replacement of a media does 
make more sense than scaling a media "endless" down. 

It should be noted that 

a) the media can only be scaled if the media itself is scaleable (that is a requirement), 
20 b) the default behavior differs from the default behavior specified in SMIL, and 

c) it is stated that the currently selected media is scaled to fit into the region's dimension, 
but by default the region f s dimension does not influence the selection of the media. 



25 



'fill" attribute: 



This attribute is extended by the following values: 

a) "amil:noscale" means that a media displayed in a specified region is not scaled at all - 
it is centered in the regions area 502. 

b) "amihupscale" means that a media displayed in a specific region is only scaled up, not 
30 scaled down. This value can be applied for specific media, whereby the behavior for 

scaling up is better than for scaling down. 

c) "amil:downscale" means that a media displayed in a specific region is only scaled 
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down, not scaled up. 

"amil:replace" means that a media displayed in a specific region is not scaled at all - but 
the most appropriate media, according to the region's dimension and under considera- 
tion of the "current situation", is selected. : 

"amilradapt" means that the underlying AMIL player decides what the best solution is. 
The player has the most knowledge and can determine what the best decision is. Based 
on meta information about the scaling behavior of the media, it is easier to find the best 
solution. Media are "scaled" and "replaced" with regards to the "current situation". 



Note: 

a) Scaling is a quite important feature and plays an important role: 

L to fit the visual component of a media into a specific region, 
ii. to adapt the visual component of a media if the region itself is 

resized; e.g. as a result of a user interaction 404 (e.g. resizing of the 
application window). Especially in this case the replacement of a 
media would improve the visual presentation. 

Example: In the case a region is scaled down by the half, a replacement of 
an H.263 encoded media in CIF image format 600 by a H.263 encoded I 
media format 600ted with QCIF image format 600 reduces the bandwidth 
significantly. 

It is quite uncommon to use a namespace prefix as a prefix for the attribute 
values. It is to emphasize the difference to the attribute values defined in 
SMIL. 

3 . AMIL Medialtems Module 

An adaptive multimedia application has somehow to describe all possible used media. As a 
result of an adaptation decision, made by the client-side middleware, a media might be re- 
1 5 placed by a different one, with different properties. 



# 



A media item 304 is an abstraction for different kinds of media, It is distinr , 
guished between discrete media items 304 and continuous media items 304 
(the later representing time-based media). A more detailed description 700 
can be found in tr Modeling of Adaptable Multimedia Documents" 
(Interactive Distributed Multimedia Systems and Telecommunication 
Services; 4th International Workshop, IDMS r 97, Darmstadt, Germany, • 
September 1997, pp. 420-429) by S. Wirag, R. Steimnetz, and L.C Wolf 
(editors). 



The "Medialtems Module" 108 serves as a description 700 unit for all, in a specific multi- 
media application, available media items 304. The content of a media item 304 can be dif- 
ferent, dependent on the characteristics of the media item 304 itself In general, it can be . 
differentiated between the following two categories: 5 

• Media items 304 describing discrete media which have the possibility to either reference 
the source by using the attribute "src" or provide the content as a part of the media item's 
content model (which can e,g. be done with XHTML as described in "XHTML™ 1 .0: ■ 
The Extensible HyperText Markup Language. A Reformulation of HTML 4 in XML - 

1 .0" (W3C Recommendation, 26 January, 2000, http://www.w3.org/TR/xhtmll/) or 
VoxML as described in "VoxML Markup Language" (The XML Cover Pages, . . 
http://www.oasis-open.org/cover/voxML.html). 

• Media items 304 describing continuous media items 304 must reference the source by ' 
using the attribute "src". 

Independent from this characteristic, every media item 304 can have meta information 
about itself as a part of its content model. Supported languages "Resource Description 
Framework" (RDF) as described in "RDF Model and Syntax Specification" (W3C Rec- 
ommendation, 22 February, 1999, http://www.w3.org/TR/REC-rdf-syntax/) or Session 
Description Protocol Next Generation (SDPng) as described in "Session Description and 
Capability Negotiation" (IETF Internet Draft, work in progress, Bormann University Bre- 
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men, 21 November, 2001, <draft-ietf-mmusic-sdpng-03.txt>) by D. Kutscher et al. 
• i ■ • .. • ■•• • • 

4 !mediaitems" element 303: 

The "mediaitems" element 303 is the outermost element of the 
"Medialtems Module 108". It serves logically as a container for all me- 
' dia items 304 used in an AMIL document 2402. 
"id" attribute: 



,v "Medialtems ModuleM08.. .... . 

"mediaitem" element 304: 

A media item 304 is an abstraction for different kinds of media. It is 



Used according to XML as an identifier for referencing the whole 



distinguished between discrete media items 304 and continuous media 
items 304 (the latter representing time-based media). An element 
"mediaitem" 304, as a child of the element "mediaitems" 303, is used to 
describe the properties/characteristics of a media item 304, like e.g. the 
location of the content and meta information, 
"id" attribute: 



It is used according to XML as an identifier for referencing the 
media item 304 by the element "mediaitem" 304. The identifier 
"mediaitem" is used 

a) in the Medialtems Module 108, and 

b) in the Adaptation Possibility Description Module 2900. 



The latter has an attribute "idref ' referencing the specific media 

item 304 in the "Medialtems module'' 1 08. 

"type" attribute: 
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It specifies the kind of media. The attribute "type" is of type 
"medialtemAttribute- TypeType" 308. 

Thereby, the attribute "type" takes one out of the following val- 
ues: 

• "audio", for audio stream, 

• "movie", for movie stream, 

• "video", for one audio and arbitrary video streams, 

• "text", for a text document, 

• "slideshow", for a slideshow presentation (e.g. take a picture 
of a movie every sixth second), 

• "textticker", (either stream or download), or 
"amil-document". 

"src" attribute: 

It specifies the location of the of the media item 304 as a URL 
"region" attribute: 

If a media item 304 has a visual component, said visual compo- 
nent is displayed in the referenced region 502. The region 502 is 
specified in the Layout Module 106. 

As depicted in Fig. 25, a media item 304 can have multiple region 
for displaying a visual component in different regions 502. 
Content model of the element "mediaitem" 304: 

The content model of the element "mediaitem" 304 is defined in a way 
that it accepts documents (or document fragments), which are docu- 
ment instances of Extensible Hypertext Markup Language (XHTML), 
Scalable Vector Graphics (SVG), Session Description Protocol Next 
Generation (SDPng), or Resource Description Framework (RDF). 



P 26797 BP 



The content model of the element "mediaitem" 304 is defined in a way 
that it accepts documents (or document fragments), which are docu- 
ment instances of Extensible Hypertext Markup Language (XHTML), 
. Scalable Vector Graphics (SVG), Session Description Protocol Next 
Generation (SDPng), or Resource Description Framework (RDF). 



Note: 

a) Owing to the lack of expressing complex constraints 802 in the XML 
Schema language, which will be replaced by Schematron or Relax NG in 
the next version, the exact semantic of the element "mediaitems" 303 in 
conjunction with the attribute "src" is the following: 



If the attribute "src" is used the content of a media item 304 can only 
f . contain meta information expressed e.g. either by RDF or SDPng. 
• If the attribute "src" is hot used, the content of the mediaitem has to 
: contain exactly one document (SVG, XHTML, etc.) and arbitrary meta 
information expressed either in RDF or SDPng. - 

b) The type attribute does not specify in detail the exact type (mime-type) of 
an element "mediaitem" 304. Enhanced properties of an element 
"mediaitem"' 304 can be expressed by the usage of RDF or as a part of 
SDPng. 

c) An element "mediaitem" 304 accepts documents (or document fragments), 
which are document instances of XHTML 1.0, SVG, SDPng or RDF. There 
is no strict parsing of these documents as described in the document "XML 

^ - schema Part 0: Primer" (W3G Recommendation^ 2 May 2001). The entity in 
charge of reading the AMDL document 2402 assumes that the content is 
well formed XML. Documents (or document fragments) are passed directly 
to the associated player, which is in charge of handling the content. 



4. Interactions Module 

A multimedia application without any interaction mechanisms appears for the user as one 
video stream. Therefore, interaction 404 is a quite important mechanism to embrace the 
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user and adapt to its personal preferences and the mobile device characteristics. The 
"Interactions Module" 1 10 specifies what kind of control elements is used in die applica- 
tion, its location and its start-up parameters. 

"widgets" element 402: 

The "widgets" element 402 is the outermost element of the Interactions 
Module 1 10. It serves logically as a container for all widget element 
404s used in the AMIL document 2402. 
"id" attribute: 

Used according to XML as an identifier for referencing the whole 
"Widget Module" 110. 
"widget" element 404: *■■«■. 
"id" attribute: , 
Used according to XML as an identifier for referencing the wid- 
get 404. 
"type" attribute: 

Based on the value of the type the information is passed to a dif- 
ferent handler. 

Thereby, the attribute "type" takes the value "class". The specific 
class is downloaded and instantiated. For any other value, a spe- 
cific player is searched for and the reference is passed to the 
player. 

"class" attribute: 

If a specific widget 404 is of type "class" this attribute has to be 
specified, referencing the class in charge of doing the interaction 
task. 

"src" attribute: 

If a specific widget is not defined or of type "noclass", this attrib- 
ute has to be specified, referencing the "src" of the widget (For 
example, this can be an SVG document.) 
"region" attribute 502: 
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If a media item 304 has a visual component - this visual compo- 
nent is displayed in the referenced region 502. Thereby, the region 
502 is specified in the Layout Module 106. 
As depicted in Fig. 26, a media item 304 can have multiple re- 
gions 502 - for displaying a visual component in different regions 
, 502. 
"param" element 406: 

The element "param" 406 is used to specify the initial parameter of the 
widget 404. lt is like a property, specified by key/value pairs expressed . ■ . 
as attributes of this element. 

If a widget 404 is of type "class", the loaded class has to support a con- 
structor for passing all the properties, 
"key" attribute: < 
Specifies the key of the property, 
"value" attribute: 

Specifies the value of the property. 



Note: 

The difference between an interaction 404 and an "mediaitem" 304 is that a 
widget is used for navigational interaction 404; a media item 304 as defined 
above is an abstraction of a media. There is an intersection between interactive 
media like a SVG 306a application and a widget - of course a widget itself can 
be realized by a SVG 306a application. But from a logical point of view it is 
worth to distinguish between the "interaction" element 404 and "mediaitems" 
element 303. 



5. Adaptation Module or. Adaptation Description Module 

The "Adaptation Description Module" 109 is by far the most important module in AMIL. It 
describes the adaptation possibilities 1500. By contrast, the modules mentioned above have 
more a "descriptive" character. Describing how the layout 3702 looks like, the property of 
each media item 304, the interactions 404, and so far. This module describes the essential 
adaptive multimedia application by referencing elements of the other modules - especially 
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the "Medialtems Module" 1 08 and the "Interactions Module" 110. 

"adaptation" element 2901: 

The "adaptation" element 2901 is the outermost element of the "Adaptation 
Description Module" 109. It serves logically as a container for the adaptation 
description 2900 used in an AMIL document 2402, 
"id" attribute: 

According to XML, it is used as an identifier for referencing the whole 
Adaptation Possibilities Description Module... 
"par" element 702: 

A "par" element, short for parallel, defines a simple time grouping in which 
multiple elements must be played back at the same time. It serves as a con- 
tainer wherein all direct child elements are started at the same time, 
"seq" element 2904: 

The "seq" element 2904 defines a sequence of elements in which elements 
play one after the other. The syncbase of the child elements of a "seq" ele- 
ment 2904 is the active end of the previous element Previous means the ' 
element which occurs before this element in the sequence time element. For 
the first child of a sequence (i.e. where no previous sibling exists), the im- 
plicit syncbase is the beginning of the sequence time element. 
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"choose" element 704 

The choose element allows specifying various alternatives. One is selected 
during playing back the multimedia presentation based on the "current situa- 
tion" of the environment Means that the characteristics of each media is ag- 
gregated once, before the media is used, and based on the environmental 
fluctuation (variation of the resource consumption), which are continuously 
monitored during playing back the multimedia presentation, one - out of the 
alternatives - is selected. 

Note: The choose element selects that media item 304 which is the best for 
the "current situation" - it does not know anything about the content, e.g. 
which scene is currently displayed. If at a specific point in time the "current 
situation" changes in the way that a continuous media item 304 is replaced by 
a different continuous media item 304 the new one is simply started accord- 
ing to a media item's "startmode" attribute. 
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Example: 
<choose> 

<mediaitem idref^"media_AJugh_quality ,f > 
<mediaitem idref^"media_Ajmiddle_quality"> 
<mediaitem idref^"media_A_low_quality"> 
</choose> 

As depicted in Fig. 30, it is assumed that the resource situation at the begins 
ning is quite good, allowing to start with a the high quality media "Afaq)" . 
While playing back, the resource condition downgrades and the presentation 
has to be adapted to the media "AOtnq)" with middle quality. Due to the fact . . 
that the resource situation further severely downgrades, the systems adapts to 
use the media "A(lq)" with the least resource consumption. After a while, the 
resource situation improves, which results in presenting "media A(hq) M 
again. The process stops - by default - as soon as the first media is finished. 
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Several elements "choose" 704: 

As mentioned above, a choose statement allows specifying different alterna- 
tives. That is quite clear if only one "choose" element 704 is considered. Li 
the more common case, however, an adaptation description 2900 consists of 
several parallel "choose" elements 704: 
<par> 
<choose> 
<mediaitem idref="ml"> 
<mediaitem idref="m2"> 
</choose> .-« 

<choose> 
<mediaitem idre5="m3"> 
<mediaitem idref="m4"> 
</choose> 
</par> 

Thereby, the enclosed element "par" 702 logically acts as a "AND" expres- 
sion. Furthermore, each elements in a "choose" element 704 is logically ex- 
pressed by an "OR" expression, This results in the Boolean expression (ml 
• OR m2) and (m3 OR m4), which may be expressed by a Disjunctive Normal 
Form (DNF) 

(ml AND m3) OR (ml AND m4) OR 
(m2 AND m3) OR (m2 AND m4). 
...... Based on mecurrentresource consumption, which is measured continuously, 

one alternative out of (ml, m3), (ml, m4), (m2, m3), or (m2, m4) is selected 
and played back. 
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Fig. 9 depicts the adaptation possibilities 1500 resulting from the adaptation description 
2900 example shown above. It should be noted that this table does only, show one property; 
in fact, various properties can be associated with one adaptation possibility 1 500. 

The adaptation description module 2900 provides the "choose" element with different at- 
tributes for modifying its default behavior. These are discussed in the following. . 

"startmode" attribute: 

This attribute specifies at which playtime a specific media, especially con- 
tinuous media, is started after an adaptation has been done* It is distinguished 
in general between the media time axis and the time axis of the enclosed 
choose element In the following the semantic of the different values are ex- 
plained by the means of the following example: 

<choose startmode="attributevalue"> 
<mediaitem idref="A"> 

<!— continuous mediaitem— > 
<mediaitem idre£="B"> 

<!— continuous mediaitem— > 
<mediaitem idref="C"> 
<!— discrete mediaitem— > 
</choose > 

Thereby, the media items 304 referenced by "A" and "B" are assumed 
to be continuous media items, and "C" is assumed to be a discrete one 
like an image. 



• 
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In Figs. 31 to 34: 

• A(tx) means that the time axis of the specific media item 304 is 
used, . ... 

• A(t) means that the time axis of the enclosed choose element is 
used. 



'A* means that the specific media item 304 is a discrete one, which 
does not have a time axis. 



restart: 

resume: 

(default) 

laststop 



playtime 



contplay- 
time 



Indicates that the media item 304 should always start 
from the beginning (cf. Fig. 31). 
Indicates that the media items 304 should always start 
from the position it stopped (cf. Fig. 32a). 
Indicates that the media item 304 should always start 
at the media time the last continuous media item 304 
contained in the same choose element stopped (cf. 
Fig. 32b): 

Indicates that the media item 304 should always start 
at the time, which is the combined playtime of all 
media items 304 contained in the element "choose" 
704 since the element "choose" 704 is started (cf. Fig. 
33). 

Indicates that the media item 304 should always start 
at the time, which is the combined playtime of all 
continuous media items 304 contained in the element 
"choose" 704 since the element "choose" 704 is 
started (cf. Fig. 34). 



"starttime" attribute: 
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Specifies the point in time when the media is started. By default, a me- 
dia starts from the beginning (see "startmode" attribute above). This 
behavior can be modified with the attribute "starttime". The attribute's 
value consists of a number and is interpreted as seconds. 

For example, the following code fragment illustrates that each media 
item 304 is started not at the beginning but rather five seconds later. 

<choose startmode="attributevalue" starttime="5s"> 
<mediaitem idref="A ,f > <! --continuous mediaitem— > 
<mediaitem idref="B"> <!-continuous mediaitem— > 
<mediaitem idref="C"> <! -discrete mediaitem- > 

</choose > 

It should be noted that this attribute is only considered, if the 
"startmode" attribute exists and its value is "restart". This constraint 
802 can not be expressed in XML Schema, 
"onremove" attribute: 

This attributes specifies what happens if a continuous media item 304 
is played back. Regarding the life-cycle of such a media item, a media 
item 304 is started, played back, the time for playing back might be 
different from the length of the movie, and is stopped if the media is 
over. This results in "stopping" or "destroying" all adaptation possibili- 
ties 1500, or simply "removing" the specific media item 304. If a 
"choose" element 704 does not contain any continuous media item 304, 
the attribute value is ignored. 

destroy Indicates that the complete choose should be removed 



this was the last remaining child in the choose. 

"looping" attribute: 



when a child is removed. 



(default) 



remove 



Indicates that the child which reached the end of its 
playtime is removed. The choose is only removed if 



m 
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Indicates that the media items 304 (if they are continuous) should start 
from the beginning if the media ends. 

true Indicates that the media is played from the beginning 

if the media ends, 
false (default) Indicates that the media is not played again, 
"evaluation" attribute: 

Specifies if the element "choose" 704 is evaluated once at start-up time, 
repeatedly in a specific time period or continuously while playing back 
the multimedia presentation. 

start-up Means that the choose element is evaluated once, 

based 6n the adaptability parameters at start up time. 

10s Indicates that the choose element is evaluated every 

specified time period, based on the adaptability pa- 
rameter at that time. 

con (default) Indicates that the element "choose" 704 is evaluated 
if a specified threshold of a parameter, describing part 
of the current situation, is exceeded. This means, that 
an adaptation will take place if the current situation 
changes, 
"priority 1400" attribute: 

Specifies the priority 1400 of the "choose". 

"empty" attribute: 
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By default the element "choose" 704 allows specifying different alter- 
natives. One of them is selected during playing back the multimedia 
presentation. If the attribute empty is set to "true" it might be, depend- 
ent on the current situation, that no media is selected out of the choose 
element. 

<choose empty="true"> 

<mediaitem idref="ml"> 

<mediaitem idref="m2"> 
</choose> 

(Results in playing back, at a specific point in time, "ml " or "m2" or 
none of both.) 

true Indicates that the alternative, that no media is selected 

out of the adaptation possibilities 1 500, is valid. 

false (default) Indicates that one media has to be selected out of the 
described adaptation possibilities 1500. 

"mediaitem" element 304: 

The element "mediaitem" element 304 specifies the used media from the 
Medialtems Module 108, which means that the media item 304 in the adap- 
tation description module 2900 is simply a placeholder. Characteristics of a 
continuous media and to some extend content of discrete media is described 
in the Medialtems Module 108. Due to the distinction between Medialtems 
Module 108 and the Adaptation Description Module 109, the description 700 

of the adaptation is quite easier. 

"priority" attribute: 

Specifies the priority 1400 of the "mediaitem" element, 
"idref ' attribute: 
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The attribute "idreF is of type "IDREF" as described in "Extensible 
Markup Language (XML) 1.0" (Second Edition, 
http://www.w3.org/TR/REC-xml) by Tim Bray (Textuality and 
Netscape), Jean Paoli (Microsoft), C. M. Sperberg-McQueen 
(University of Illinois, Chicago), and Eve Maler (Sun Microsystems, 
. Inc.), referencing a specific "mediaitem" part of the "Medialtems Mod- 
ule" 108. The link realize the dependency "Linking Mediaitems". 
"starttime" attribute: 

Specifies the point in time the media is started. By default the media 
starts from the beginning (see "startmode" attribute above). This be-, 
havior can be modified with the attribute "starttime". The attribute's 
value consists of a number and is interpreted as seconds. 
For example, the following code fragment illustrates that the media 
item 304 is started not at the beginning but rather five seconds later. 

<mediaitem idref="A" starttime^'Ss'^ 
It should be noted that this attribute overrides the attribute "starttime" 
from the element "choose" 704. It is only considered in the following 
two cases: 

1 . if the media item 304 is a part of a "choose" element 704, and the 
attribute "startmode" is specified there and has the value "restart", 

2. if the media item 304 is not part of a "choose" element 704. 

In the later case, the value specifies the initial start time of the media 
item 304. 



In the following, the "switch" element used in SMIL shall be compared with the "choose" 
element 704 of AMIL according to one embodiment of the underlying invention. 

a) From a semantic point of view 



A choose element in SMIL allows "an author to specify a set of alternative elements from 
which only one acceptable element should be chosen. An element is acceptable if the ele- 



• 
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ment is a SMIL element, the media-type can be decoded, and each test-attribute of the ele- 
ment yields "true". An element is selected as follows: The player evaluates the elements in 
the order in which they occur in the switch element. The first acceptable element is selected 
at the exclusion of all other elements within the switch. 

This is in contrast to the semantics of the element "choose" 704 used in the "Adaptation 
Description Module" 109. As described above, all possible media of an element "choose" 
704 are evaluated, which means that the media's properties (e.g. bandwidth, duration, etc.) 
are evaluated and stored in an internal data structure - cf. description of the "Configuration 
List" in "Modeling of Adaptable Multimedia Documents" (Interactive Distributed Multi- 
media Systems and Telecommunication Services; 4th International Workshop, IDMS'97, 
Darmstadt, Germany, September 1997, pp. 420-429) by S. Wirag, R. Steinmetz, and L.C. 
Wolf (editors). At start-up time and if the "current situation" changes, the adaptation de- 
scription 2900 is evaluated and the most appropriate adaptation possibility 1500, for the? 
"current situation", is selected and played back. 

b) From an SMIL document author point of view 

In SMIL, the author describes which media is used in which condition. The condition 
(logical expression described with the attribute "test") is a fix part of each media element in 
SMIL. During start-up time, the SMIL player sequentially evaluates each media element's 
condition. If the condition can be fulfilled, the media is selected for playing back. . 

Note: 

a) The author specifies which resource are considered (e.g. bandwidth, screen size, 
etc.) 

b) The author specifies the order in which the elements" condition is evaluated. 

c) The current situation, as used in SMIL, is described by the properties used in the 
conditions. 

d) The SMIL standard does not claim, that the element "switch" is re-evaluated if the 
"current situation changes. 

e) The focus of SMIL is on applications running in a static environment. The applica- 
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tion is adapted once, during start-up. 

In SMIL, the properties describe the situation in which one or a set of media items 304 has 
to be chosen. In the "Adaptation Description Module" 109 the properties describe attributes 
of the media items 304 and the "AMIL player" uses this information to find out which set 
of media items 304 to use for an actual situation. (Example: If the available bandwidth is 
below 1 MBit/s, two media items 304, A and B, are used. A needs 300 kBit/s and B needs 
400 kBit/s.) In SMIL, the author has to manually create the mapping from media attributes 
to situations during authoring time. In the "Adaptation Description Module" 109 this map- 
ping is left to the browser, which simplifies the authoring significantly, especially as the 
media item properties can often be created automatically, thereby using meta data for the 
media or by examination of the media files. 

In AMIL, the "AMIL player" aggregates information about the "current situation" and the 
media, which is apart of the adaptation; description 2900. Based on the "current situation", 
that is monitored continuously, a specific "adaptation possibility" 1500 is selected and used 
for playing back until the "current situation" changes and a new "adaptation possibility" 
1500 is selected. The author can give hints to express the author's preferences, which can 
influence the decision which adaptation possibility 1500 is used for playing back. In AMIL, 
the decision which media is chosen is determined by the adaptation control unit, but the 
author of the document can influence the adaptation control process by the usage of attrib- 
utes supported by elements of the "Adaptation Possibility Module" 109 - e.g. the priority 
attribute 2810 of the elements "switch" and "mediaitems" 304. 

c) From an adaptation point of view 

If the "current situation" changes, an adaptation is accomplished, which means that media 
are substituted. The exact behavior how this is done is not described in SMIL. AMIL sup- 
ports the author of a document with different attributes, e.g. a "startmode" for specifying 
the transition from one media to another media, restart and looping functions if the movie 
is over, and so on. 
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Priority attribute i > 

The attribute "priority" 2810 is supported with the elements "choose" 704 and Vmediaitem" 
304 and enables the weighting of adaptation possibilities 1500. To understand the idea of 
5 priorities 1400, fee concept of adaptation possibilities 1500 should be reminded. 

Based on the "Adaptation Possibility Description" mentioned above, different adaptation 
possibilities 1500 are calculated. While playing back, appropriate adaptation possibilities 
1500 matching the "current situation" are selected. Thereafter, the adaptation possibility - 
10 1 500 with the highest priority 1400 out of the set of appropriate adaptation possibilities 
1500 is selected. 

The first activity diagram depicted in Fig. 23 shows that if the "current situation" has 
changed; out of all available adaptation possibilities 1500 the set that matches the "current 
15 situation" is filtered out. Out of this set, the adaptation possibilities 1 500 with the highest 
priority 1400 are selected. If at the end more than one adaptation possibility 1500 is left, 
either that with the lower resource consumption is selected or the selection is done ran- 
domly. 

20 The evaluation of the priorities 1400 is done in line with the calculation of all adaptation 
possibilities 1500. Fig. 15 shows the adaptation possibilities 1500, thereafter sometimes 
abbreviated with "AP", derived from the adaptation possibility 1500 description 700 de- 
picted in Fig. 13. Adaptation Possibilities (APs) are calculated in such a way that a 
"choose" element 704 is considered as an "OR" operator, and a "par" element 702 as an 

25 "AND" operator. This results in the following "Boolean" expression: 

(M4 AND M2) OR (M4 AND Ml) OR 
(M3 AND M2) OR (M3 AND Ml). 

30 Thereby, the selection of one "AND" expression (e.g. M4 AND M2) (also named adapta- 
tion possibility 1500) is based on the "current situation". 
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The evaluation of the associated priority 1400 of an adaptation possibility 1500 is done in 
the following way: 

1 . All children of a "par" element 702 are sorted according to their priority 1400. 
5 2. The configurations of the first two child elements are merged (by means of an AND 

operator). The priority 1400 of the resulting configurations consist of the priority 1400 
of the higher prioritized child is appended to the priority 1400 of the lower-prioritized 
child. (Thereby, the character "-" is used as a separator.) 

3. The result is repeatedly merged with all other children of the* "par 1 ? element 702. The 
10 priorities 1400 are always handled as in step 2. 

4. Element of type "choose" 704 do not perform any priority calculation. The element 
"choose" 704 represents an "OR" operation, in which the results from the child ele- 
ments are just combined and not modified - including the priorities 1400. 

15 The result at the end is a list of possible adaptations attributed with an attribute priority 
1400 calculated from the priorities 1400 of the used media of this possible adaptation. 

Linking between elements 

20 The term "link", as used in this document, describes the connection of different elements in 
one AMU, document 2402 - in contrast to the general meaning where various Web re- 
sources (in different documents) are linked together. A link is described by its "source an- 
chor". 2702 and its "destination anchor" 2704 as depicted in Fig. 27. An anchor's source 

... •*.'-**■. 

2702 is described by the usage of the XML attribute "id" - an anchor's destination 2704 by 
25 the XML attribute "idref \ Both attributes are validated during the parsing of an XML 
document (without the usage of XML Schema). 

Fig. 5 depicts the association between various modules. Linking is quite important since 
the information itself is distributed over different modules. The "Adaptation Description 
30 Module" simply describes the adaptation possibilities 1500 - the media itself is described in 
the "Medialtems Module" 108, interactions 404 are described in the "Interactions Module" 
110, and the whole layout 3702 is described in the "Layout Module" 106. Said modules 
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enable the splitting of information, which allows the modification of e.g. the layout 3702 
without touching anything from the "Medialtems Module" 108 or the "Interactions Mod- 
ule" 1 10 - as long as all linked region are still available. 

5 Fig. 9 depicts the name of the association and their modules used to describe the linking 
1600 between the AMEL modules. 

Fig. 16 shows an exemplary instance document For the sake of that example, the docu- 
ment is hold quite simple. Thereby, the adaptation description 2900 is reduced to one me- 
10 dia item 304, the properties of that media item 304 are described in the "Medialtems Mod- 
ule" 108 and linked with the aid of the attribute "idref f . Both elements, the media item 304 
of the "Medialtems Module" 108 and the widget 404 of the "Interactions Module" 1 10, . 
refer to an element region 502 of the "Layout Module" 106 where they are displayed. Fig. 6 
shows how this multimedia presentation looks like. 



Note: 

a) A "mediaitem" element 304 of the "Medialtems Module" 108 is not needed to 
support a reference to an element "region" 502 of the "Layout 3702 Description 
Module". This is optional and makes only sense for media with a visual compo- 
nent, which means that if a media specifies a region for rendering and it has a 
visual component, this visual component is displayed in the* referenced region. 

b) The visual component is set to the size of the specified region. If the media does 
not fit into the region it is scaled by default. * 



Granularity of reuse 

As authoring of multimedia presentations is a quite time-consuming and costly task, atten- 
20 tion has been drawn to reuse adaptive multimedia applications for efficiency and economi- 
cal reasons. However, this has to be considered quite carefully. First, the documents are 
generated dynamically for each request. Therefore, it is quite hard to reference a part of 
document which is not statically available. 
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Moreover, there is no restriction in using a static AMIL document 2402 saved e.g. on the 
file system. For this reason, a further media type named "amil-document" has been intro- 
duced. A media item 304 of type "amil-document" is rendered in its specified region. 

5 Fig. 17 shows how an adaptive multimedia presentation can be used as part of another one. 
Thereby, the multimedia application "presenstation-B.amil" is referenced as a media item 
304 out of another multimedia application described in "presentation- A.amil ,f . 



Note: Several elements contain an attribute of type M ID", which allows refer- 
encing them. The current version of AMIL, as described here, does only sup- ; 
port the reuse 1700 (referencing) on document level. Further version might 
allow the reuse 1 700 (referencing) of specific modules. 



10 AMIL namespace 

Namespaces (see World Wide Web Consortium: "Namespaces in XML", 14 January, 1999, 
http://www.w3.org/TR/REC-xml-names/) are quite common in the XML world to distin- 
guish between different vocabularies. AMIL use the URI "www.sony.de/amil" to reference 
15 to the newest version of the vocabulary. Fig. 40 shows a complete AMIL document 2402, 
wherein the namespace is explicitly used throughout the whole document. 

AMIL Document Object Model (AMIL DOM) . 

20 In the scope of the underlying invention, an AMIL Document Object Model (AMIL DOM) 
has been introduced to describe an AMDL document 2402 in an abstract document model 
structure. It supports the dynamically access and update of the content and structure of an 
AMEL document 2402. The document can further be processed, and the results of that 
processing can be incorporated back in the presented document. The AMIL DOM is quite 

25 straightforward. In fact, it is an extension to the DOM proposed by W3C. Like "XHTML 
306b DOM" and "WML DOM", "AMIL DOM" simply extents the basic W3C DOM by 
means of an AMIL-specific vocabulary. 
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In extension to state-of-the-art document object models, AMDL DOM supports a simplified 
transaction model. This allows making a set of logically grouped manipulations to the 
AMDL DOM which results in the support of transactional modification for adaptive docu- 
ments. For example, the replacement of an "adaptation-description-specific" element - 
"choose" 704 has to be done as an atomic change, which results in a recalculation of all 
adaptation possibilities 1500. If that is not done atomically, a modification might be a quite 
crucial step, which results in an inconsistent state. 

As depicted in Figs. 28 and 35, the AMIL DOM supports, aside from state-of-the-art 
primitives, primitives for doing simple transaction-oriented processing by enhancing the 
interface "document" 2806 with a "commit" primitive. It should be noted that modifica- 
tions are not passed to the middleware until the "commit" ^ primitive is triggered. 

After having created an internal presentation of an AMIL document 2402, it can be adapted 
to changes in the "current situation", e.g. a user interaction 404 or the change of the user's 
location. Modifications do not get visible until they are committed. This fact is depicted in 
Fig. 36. The illustration on the right shows how the layout 3702 can change after the 
document has been modified. 

The following sections introduce modules that can optionally be used for a more detailed 
description 700 of the adaptation process. They are not mandatory, which means that an 
AMIL document 2402 is semantically valid without these modules. 

1. Events Module 

An "event" 3802 is a signal to the AMIL Browser from any sensor, indicating that the 
"current situation" has changed. As mentioned in the introduction, the "current situation" is 
categorized into three different dimensions: network QoS, mobile device capabilities and 
user context. Li this scenario, an AMIL document 2402 is dynamically generated at request 
time, based on the "current situation". After that, the supporting middleware, as described 
in the article "Modeling of Adaptable Multimedia Documents" (Interactive Distributed 
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Multimedia Systems and Telecommunication Services; 4th Intemanonal Workshop", 
IDMS'97, Darmstadt, Germany, September 1997, pp. 420-429) by S. Wirag, R. Steinmetz, 
and L.C. Wolf (editors), controls the adaptation process of the media specified in the AMU, 
document 2402 - but the AMIL document 2402 remains quite static. With the introduction 
5 of the AMBL "Events Model" 118, changes of the "current situation" can influence the 
document in different ways: 

a) Modification of the linking structure: The AMIL document 2402 consists of a vo- 
cabulary, a hierarchical structure (which defines how the elements are nested) and a 
non-hierarchical structure (which describes how the elements are linked). A change 
of the "current situation" can result in the modification of the linking structure. 
Thereby, the associations "Linking Mediaitems and Layout" and "Linking Widgets 
and Layout" are changed (cf. Fig. 37). : 

b) Modification of the whole document structure: This results in an updated or totally 
different AMIL document 2402. 

Fig. 38 shows some types of possible events 3802. In detail, each parameter, describing 
10 part of the "current situation", may be modeled as an event 3802. Fig. 24 shows the influ- 
ence of an event 3802 to a specific instance of an AMIL Document. In case a) the internal 
linkjhg structure of the document is changed, in case b) the essential hierarchical document 
structure and the internal linking 1600 are changed; this might result in a updated or totally 
different document. 



Note: 

a) By the time the. invention h.as been madp,.the focus was on "modification of the 
linking structure". By contrast, "modification of the whole document" has not 
been considered so far. However, modification of the whole document is a quite 
interesting topic if the "conversational" and "document retrieval" scenario are con- 
sidered together. It should be noted that, based on changes in the "current situa- 
tion", the whole document structure might be changed. 

b) The change of the "current situation" can be forwarded to the AMDL Browser 



(which is in charge of doing the document processing) in a way described in this 
section and the following subsection. But an event 3802 simply says that some- 
thing has happened, more detailed information about the event 3802 is not passed. 
Therefore, if necessary, additional functionality 1800 has to be provided as , 
part/extension to the XPath and XSLT specification for use in expressions. (As 
shown in the following subsection, XPath and XSLT are used for describing 
"conditional" expressions in an "action".) 



Fig. 1 8 shows a conditional statement, which might be part of an action element. The func- 
tion "diff" is an extension which is used to describe that the content of the test element 
should be evaluated if the difference between the "currentlocation" and the location at the 
point ("47,1 1","9,34") (which denotes longitude and latitude, respectively) is less than 500 
meters. 

The Events Module 118 supported by AMDL allows acting on changes of the "current 
situation". The module itself is optional and has not to be supported/ 

"adaptationevent" element: 

The "adaptationevent" element is the outermost element of the Adaptation , 
Events Module 1 1 8. It logically serves as a container for the adaptation 
events description 700 used in an AMDL document 2402. 
"id" attribute: 

It is used according to XML as an identifier for referencing the whole 

Adaptation Possibilities Description Module, 
"listener" element 3906: 

The "listener" element realizes a binding of an event 3802 to a specific ac- 
tion, which is the only child a listener have, 
"type" attribute: 



* 
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The attribute ''type" specifies which event 3802 the listener 3906 is 
listening for. It is simply an identifier, reflecting that a property of a 
specific sensor has changed. There is no additional information avail- 
able which is associated with the event 3802. 

Note: •> . ' ••■ 

a) The set of supported events 3802 is described in the XML 
Schema "EventType" as an enumeration. This should not scare 
of defining additional event 3802 types. The "EventType" is de- 
fined in XML Schema and can domain-specifically be ex- 
tended. 

- b) The (namespace-specific) attribute value has to be conform to 
the following pattern: 

"sensor" :: "name of the event" 

Thereby, the sensor specifies the source of the event 3802, and the 
f name specifies the kind of the event 3802. The pattem is required, and 

the validation, process checks the rule, 
action" element 3910: . 

The "action" element 3910 describes the activity as a result to an event 3802. 
The content of the element consists of "conditional" and "action" statements 
(e.g. the "link" statement). Due to the fact that a vocabulary for expressing 
conditional statements is already available in XSLT, it is taken over and in- 
tegrated into the Events Module 118. This means that statements applied to 
the document (e.g. the "link" statement) can either be used in an conditional 
statement, to consider the change of the "current situation" in more detail, or 
directly. Fig. 19 shows ah exemplary document fragment of an adaptation 
event description 1900/2000 using XSLT's conditional statements. Fig. 20 
depicts a document fragment describing an action without the usage of 
XSLTs conditional statements.' 
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"link" element 3914: 

The "link" element 3914 changes the linking structure of a document. It re- 
places the link described in the attribute "element". 

Note: Hiis attribute is of type "BDREF", which means that their existence is 

validated during reading the AMIL document 2402. 

"element" attribute: 

Specifies the element, by its unique "id", which has a link to be modi- 
fied. 

"source" element 3916: 

The "source" element 3916 specifies the source of the link, 
"ref attribute: 

It specifies the name of the attribute, and which value has to be modi- 
fied. The attribute itself is the source of the link, 
"target" element 3918: 

The "target" element 3918 specifies the new target of the link. 

"ref attribute: 

It specifies the new link target. 
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Example: If in the following code fragment the link between the media item 304, 
uniquely addressed by the attribute "id", and the region 502, uniquely addressed by 
the attribute "id", 

<region id="regionA" .../> 
<region id^'regionB" ..J> 

<mediaitem id="mediaitemA" region="regionA"> 

■ should be modified, the "source" element 3916 and the "target" element 3918 have 
to be written as follows: 

<link element=="mediaitem ,, > 

<source ref="region" 502/> 
<target ref="regionB ,, > 
</link> 

This results in that the mediaitem's visual component uses the "regionB" instead of 
the "regionA". The exact behavior of the media's visual components depends on the 
region's attribute "fit". * 

2. Constraints Module 

The Constraints Module 1 16 allows adding additional constraints 802 to the adaptation 

, . — • »• . . •• • %.» • 

5 description elements to influence (in most cases to reduce) the set of possible adaptation 
alternatives 2100. When constraints 802 are applied, it is e.g. possible to remove all adap- 
tation alternatives 2100 (each consisting of different media items 304) that need a higher 
"bit rate" than specified by the respective constraint 802. 

10 Thereby, it is useful to allow such constrains also for the "mediatiem" element 304 in spite 
of the fact that there are no alternatives on this level. This is because a media item 304 may 
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be able to run in different media configurations. A media item 304 maiy expose different 
media configurations, which can dynamically be selected when playing the media item 304. 
Each media configuration may result in different media properties like different quality, 
resource requirements or error robustness. Examples for such media items 304 are video 
5 streams using a layered codec, a live stream whose coding parameters can be influenced 
from the receiver side or media items 304 that can dynamically apply Forward Error Cor- 
rection (FEC) algorithms. When constraints 802 are used, the author has the possibility to 
exclude unwanted media configurations. 

10 It is important to understand the difference of the semantics between a constraint 802 ap- 
plied to a media item 304 and another adaptation description element When applied to a 
media item 304, it restricts the possible media item configurations. When applied. to an 
adaptation description element (e.g. like "choose" 704), it applies to all possible adaptation 
possibilities 1500 of that element (each consisting of a set of media items 304). In the 

15 AMBL code depicted in Fig. 21, the outermost "par" has at least four adaptation alternatives 
2100 (ml AND m3, ml AND m4, m2 AND m3, m2 AND m4). If the media items 304 
allow different media configurations, the list is further extended as two alternatives either 
differ in one media item 304 or one media configuration. 

20 A constraint 802 applied to this "par 11 element 702 could remove one or more of these al- 
ternatives. For example, a constraint "bitrate < 500 kBit/s" would remove all alternatives 
whose media-items-combined bit rate is higher. 

The author has to take care that the constraints 802 are consistent and do not result in a 
25 situation, where all adaptation alternatives 2100 are removed*. — 

In the following section, the "Constraint Module" 800 supported by AMTT. language shall 
be introduced. 



• 
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"constraints" element 801 : 

The "constraints" element 801 is the outermost element of the Constraint 
. Module, It logically serves as a container for all "constraint" elements 801 
, used in the AMIL document 2402. 
"id" attribute: 

It is used according to XML as an identifier for referencing the whole 
"Constraint Module" 800. 
"constraint" element 802: 

; The "constraint" element 802 is a container for several "restriction" elements 
804, "adaptationtrigger" elements 806 and/or "adaptationfrequency" elements 
808. It can be referenced by other elements that want to apply the defined con- 
straint 802. 
"id" attribute: 

Used according to XML as an identifier for referencing the whole 

"constraint" 802. 
"restriction" element 804: 

The "restriction" element 804 is used for describing the actual restrictions im- 
posed by a constraint 802. 
, "value" attribute: 

The "value" attribute contains a textual description 700 of the actual re- 
striction. 1st value consists of different conditions. A condition itself con- 
sists of a "restriction identifier" followed by a comparison operator, fol- 
lowed by a value plus optionally a unit (example: "bandwidth < 500 
kBit/s"). Conditions can be combined using the "AND" operator or the 
"OR" operator. The restriction identifier determines the possible com- 
parison operators,,the value range and the unit. Fig. 1 1 depicts the sup- 
ported restriction identifier. 

The advantage of having this data in an attribute value and not in several 
XML elements or attributes is that additional restriction identifiers can be 
added later on without the need to change the AMIL grammar. 



797 BP 

"adaptationtrigger" element 806: 

The "adaptationtrigger" element 806 is used for describing conditions where a 
media item 304 is considered to have a problem and therefore an adaptation 
should be performed if the condition becomes true. 
It is important to note the difference between a "restriction" 804 and an 
"adaptationtrigger" 806. Whereas a "restriction" 804 limits the number of pos- 
sible adaptation alternatives 2100, an "adaptationtrigger" defines conditions 
that should not occur during the playtime of a media item 304. 
"value" attribute: 

The value attribute contains a textual description 700 of the actual adap- 
tation trigger. Its value is always a string consisting of different condi- 
tions. A condition consists of a condition identifier followed by a com- 
parison operator, followed by a value plus optionally a unit (example: 
"framerate < 15 frames/s"). Conditions can be combined using the 
"AND" operator or the "OR" operator. The identifier determines the pos- 
sible comparison operators, the value range and the unit. 

Fig. 12 shows the supported "Adaptation Trigger Identifier". 

To evaluate the adaptation triggers, the AMIL player needs a means to 
measure the corresponding values during playtime of the media item 304. 
If the AMIL player does not have a possibility to measure a certain value 
or the specified condition does not apply to the media item's type, the 
corresponding adaptation trigger is ignored. 

The advantage of having this data in an attribute value and not in several 
XML elements or attributes is that additional restriction identifiers can be 
added later on without the need to change the AMBL grammar. 
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"adaptationfrequency" element 808: 

The element "adaptationfrequency" 808 is used for describing how often ad- 
aptations should occur. Adaptations are always necessary if the situation gets 
worse. If the situation improves, different strategies can be applied which de- 
termine how fast to react. This results in either a more "stable" adaptation be- 
havior or an "agile" adaptation behavior. This behavior can be influenced us- 
ing this element. • - 

The element "adaptationfrequency" 808 does not have a meaning if associated 
to a "mediaitem" element 304 or another element, which does not have any 
adaptation possibilities 1500. An "adaptationfrequency" element 808, which is 
associated to an adaptation description element, overwrites all other "adapta- 
tionfrequency" elements 808 which are associated to elements inside this ad- 
aptation description element. 

The defined adaptation frequency always applies to all adaptation alternatives 
2100 of the adaptation description 2900 element it is assigned to. Therefore, 
one part of the adaptation description 2900 can be set to react "agile" and an- 
other part to react "stable". The adaptation algorithm will then try to select 
adaptation possibilities 1500 during runtime, in which the "agile" parts are 
changed more often than the "stable" parts, 
"value" attribute: 

This attribute specifies how the adaptation behavior should be. The pos- 
sible values are "stable", "moderate" and "agile" 

Fig. 1 1 gives an overview of the restriction identifiers contained in the "value" attribute of 
the "restriction" element 804. Possible comparison.operators are: <, >, <=, >=. • - - 



Fig. 12 gives an overview of the adaptation trigger identifiers included in the "value" at- 
tribute of the "adaptationtrigger" element 806. Possible comparison operators are: <, >, <=, 
and >=. 
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Claims 

1 . A method for describing adaptive mobile multimedia applications and/or presentations, 
5 whose playback behavior inherently depends on the current situation at runtime, . 

the method being based on an XML-based document model (100) and comprising the step 
of 

- describing the intrinsic adaptation possibilities of application and/or presentations, which 
run in a mobile network environment, in an Adaptation Module (109) comprising the vo- 

10 cabulary and language structure required for describing the adaptation possibilities of said 
adaptive mobile applications. 

2. A method according to claim 1, .. 
furthermore comprising the step of ' 

15 - describing the available media for the application and/or presentation in a Medialtems 

Module (108) comprising the vocabulary and language structure required for describing the 
media items (304) used within said adaptive mobile applications. 

3. A method according to anyone of the preceding claims, 
20 furthermore comprising the step of 

- describing the interaction parameters of the application/presentation in an Interactions 
Module (110) comprising the vocabulary and language structure required for describing the 
interaction possibilities (404) used for said adaptive mobile applications. 

25 4. A method according anyone of the preceding claims, 
furthermore comprising the step of 

- describing the constraints of the adaptation process in a Constraints Module (116) com- 
prising the vocabulary and language structure required for describing constraints (802) for 
said adaptive mobile application. 

30 

5. A method according anyone of the preceding claims, 
furthermore comprising the step of 
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changing the language structure and vocabulary of the modules (106, 108, 109, 110, 116) 
in an Events Module (118) comprising the vocabulary and language structure required for 
describing the event possibilities used in said mobile applications. 

6. A method according to anyone of the preceding claims, 
furthermore comprising the step of 

- describing the association between the Adaptation Module (109) and the Medialtems 
Module (108), represented by a link. 

7. A method according to anyone of the preceding claims, 
furthermore comprising the step of 

- describing the association between the Adaptation Module (109) and the Interactions 
Module (1 10), represented by a link. 

i 

i 

8. A method for describing a XML-based document serving as a connection layer between 
a middleware framework supporting mobile adaptive multimedia applications and an 
authoring system supporting the generation of mobile adaptive multimedia applications 
(structure and vocabulary) as described in anyone of the claims 1 to 9, 

characterized in that the description is carried out by means of a language comprising: 
_ at least one Medialtems Module (108) serving as a description unit for available media 
items (304) within said multimedia applications, 

- at least one Layout Module (106) which organizes said media items into regions (502) 
on the visual rendering surface of a mobile display device, and 

- at least one Adaptation Module (1 09) which controls a context-aware adaptation of said 

- distributed multimedia applications by referencing elements of the Medialtems Module 
(108). 

9. A method according to claim 8, 
wherein the language furthermore comprises: 

- at least one Constraints Module (1 16) which allows adding additional constraints (802) 
to the adaptation description elements (2900), and 

- at least one Events Module (118) which allows to react on changes of various resources 
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encompassing user's physical environment (location, temperature), user's context, qual- 
ity-of-service (QoS) conditions of the applied netwoiks, and mobile device capabilities. 

10. A method for operating a middleware framework supporting the processing of an 
XML-based description of an adaptive mobile application/presentation (100) according to 
anyone of the claims 8 and 9, 

characterized in that 

said middleware framework allows each running mobile multimedia application to specify 
the media it wants to use and the relationships between these media, calculates the adapta- 
tion possibilities (1500) of mobile multimedia applications and controls the adaptation 
process in dependent on the current situation. 

1 1 . A method according to claim 10, 
characterized by 

the step of modifying the linking structure between the Adaptation Module (109) and the 
Medialtems Module (108) in case the current situation has changed* 

12. A method according to claim 10, 
characterized by 

the step of modifying the linking structure between the Medialtems Module (108) and the . 
Layout Module (1 06) in case the current situation has changed. i 

13. A method according to claims 10, 
characterized by 

of modifying the linking structure between the Interactions Module (110) and the Layout 
Module (1 06) in case the current situation has changed. 

14. A method according to claims 10, 
characterized in that 

the document linking structure and/or the document structure and document vocabulary 
itself are modified by user interactions (404). 
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15. A method according to claim 14, 
characterized in that 

user interactions (404), which are used to modify the document linking structure and the 
document itself; are described by the Interactions Module (1 10). 

16. A method according to anyone of the claims 10 to 13, 
characterized by 

dynamically binding media items (304) to a specific region (502) on the visual rendering 
surface of the mobile display device with the aid of the Events Module (118), initiated by 
changes (3700) of the current situation. 

17. A method according to anyone of the claims 1 1 to 14, 
characterized by 1 

dynamically binding widgets (404) to a specific region on the visual rendering surface of 
the mobile display device with the aid of the Events Module (1 1 8), initiated by changes 
(3700) of the current situation. 

1 8. A method according to anyone of claims 8 and 9, 
characterized by 

the step of an extending or newly specifying at least one attribute of at least one element of 
the Layout Module (106) in order to adapt the visual component (600) of a specific media 
item (304) to the dimension of those regions (502) on the applied mobile display device 
which are intended for multimedia presentations by scaling scaling-up /scaling-down the 
visual size of said media item (304) or replacing the said media item (304) dependent on 
the current situation. ~ 

19. A method according to anyone of claims 8 and 9, 
characterized by 

describing (700) alternative media items (304) of the Medialtems Module (108) used in the 
Adaptation Module (109) by means of 

- media-specific information encompassing bandwidth and size of the visual portion of a 
multimedia presentation, 
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~ meta information encompassing the name, the genre, and the actor of the media in case 
of a media item (304) of type video, and/or 

- inline information or a reference to external resources by the usage of Universal Re- 
source Identifiers (URIs). . t 

20. A method according to anyone of claims 8, 9, 10 and 19, 
comprising the step of 

specifying various alternatives both at start-up time and in case of changes of the current 
situation by means of a choose" element (704) of the Adaptation module (109). 

21 . A method according to claim 20, 

characterized by the steps of -'./■'.■ 

- selecting the most appropriate adaptation possibility (1500) at start-up time, 

- continuously monitoring the network conditions, the available mobile device capabili- 
ties and/or the user context, and 

- selecting the most appropriate adaptation possibility (1500) in case of changing network 
conditions, mobile device capabilities and/or user context. 

22. A method according to claim 21, 
characterized by the step of 

influencing the adaptation control process by the usage of priority attributes supported by 
the respective elements of the Adaptation Module (109). 

23. A method according to claim 22, 
comprising the step of 

using a Par Element (702) of the Adaptation Module (109) for defining a simple time 
grouping in which multiple elements must be played back at the same time. 

24. A method according to claim 23, 
characterized in that 

the adaptation possibilities (1500) are calculated with the aid of a Boolean term expressed 
by a Disjunctive Normal Form (DNF) on a set of different media items (304), wherein a 
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"choose" element (704) is considered as an "OR" operator and a "par" element (702) as an 
"AND" operator, from which one conjunction of the Disjunctive Normal Form (DNF), the 
adaptation possibility (1500), is selected, depending on the quality-of-service (QoS) of the 
applied networks, the mobile device capabilities and' the user context. 

25. A method according to claim 24, 
characterized by the step of 

using an "Adaptation Description Module" (2900) which provides the "choose" element 
(704) with a i'startmode" attribute for modifying its default behavior, which specifies at 
which playtime a specific media, especially continuous media, is started after an adaptation 
has been executed due the change of the current situation. 

26. A method according to claim 25, 
characterized in that 

the continuous media item (304) is replaced by a different continuous media item (304) due 
to the change of the current situation, and the new one is simply started according to a me- 
dia item's "startmode" attribute. 

i 

27. A method according to claim 26, 
characterized in that 

me "startmode" attribute can take one of the following values: 

- a "restart" value, which indicates that the media item (304) should always start from the 
beginning, 

- a "resume" default value, which indicates that the media items (304) should always start 
- from the position it stopped, • - - - 

- a "laststop" value, which indicates that the media item (304) should always start at the 
media time the last continuous media item (304) contained in the same "choose" ele- 
ment (704) stopped, 

- a "playtime" value, which indicates that the media item (304) should always start at the 
time, which is the combined playtime of all media items (304) contained in the "choose" 
element (704) since the "choose" element (704) is started, and 

- a "contplaytime" value, which indicates mat the media item (304) should always start at 
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the time, which is the combined playtime of all continuous media items (304) contained 
in the "choose" element (704) since the "choose" element (704) is started 

28. A method according to claim 27, 
5 characterized in that 

the "Adaptation Description Module" (2900) supplies the "choose" element (704) with an 
"onremove" attribute specifying what happens after a continuous media item (304) is 
played back. 

10 29. An method according to claim 28, 
characterized in that 

the "Adaptation Description Module" (2900) provides the "choose" element (704) with an 
"evaluation" attribute which specifies if the content model of the element "choose" (7.04) is 
evaluated once at start-up time, repeatedly in a specific time period or continuously while 
15 playing back the multimedia presentation. 

30. A method according to anyone of the claims 28 and 29, 
characterized in that 

the "Adaptation Description Module" (2900) provides the "choose" element (704) with an 
20 "empty" attribute which supports the functionality that the set of media appropriate for spe- 
cific current situation can be empty. 

31. A method according to claim 30, 
characterized in that 

25 the evaluation of the associated priority (1400) of an adaptation possibility (1500) is done 
by 

— sorting all children of a par element according to their priority (1400), 

— merging the configurations of the first two child elements by means of an "AND" op- 
erator in such a way that the priority (1400) of the resulting configurations consist of the 

30 priority (1400) of the higher prioritized child appended with the priority (1400) of the 
lower-prioritized child, and 
repeatedly merging the result with all other children of the par element. 
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27. Sep. 

Abstract 

The invention generally relates to the field of markup languages used to describe adaptive 
mobile multimedia applications and/or presentations being inherently dependent on the 
dynamic mobile environment they are running in, which means that these mobile multime- 
dia applications and/or presentations need to be adapted to the preferences of mobile users, 
the capabilities of their mobile computing devices, and their current situation. It allows 
adaptive mobile stream-based multimedia applications with real-time requirements in a 
typical wireless scenario (e.g. a radio link with a changing transmission quality and hand- 
over procedures) to adaptively and responsively react to a time-varying network topology 
and different radio link characteristics. Thereby, the underlying invention especially in- 
cludes research and development issues in the field of describing adaptation possibilities 
(1500), adaptation constraints (802) and adaptation events (3802) directed to a personali- 
zation and context-aware adaptation of document-based multimedia applications by pro- 
viding methods for pre-allocating, reserving, monitoring and adapting QoS-related parame- 
ters in a dynamic mobile environment using an XML-based multimedia presentation lan- 
guage. 

In this connection, a document model (100) consisting of vocabulary, document structure 
and linking means (1600) between the document model elements is presented which sup- 
ports the description (700) of adaptive mobile multimedia applications and/or presenta- 
tions. Besides, a document object model supporting a simplified transaction-oriented ac- 
cess is proposed. 
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head 
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<xs: element name =" amil " > , 
<xs : c omp 1 exType > ' 
<xs : sequence > • 

<xs: element name«"head" type="headType"/> 
<xs: element name="body" type= s "bodyType»/> 
< /xs : sequence> 

<xs:attribute name="id» type="xs : ID" 

use= 11 required" / > 
<xs:attribute name= "version" type="xs : string" 

use= n required" / > 
</xs : complexType> 
</xs : element > 
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<xs:eletnent name= "mediaitems " type«»mediaitemsType" /> 

<xs : complexType name= "medial temsType"> 
<xs: choice maxOccurs= "unbounded" > 
<xs: element name= "mediaitem" > 
<xs : complexType > 
<xs:all> 

<xs : any 

namespace="http ://www.w3 . org/1999/xhtml ■ minOccurs= n 0" 

processContents= ■ skip"/> 
<xs : any namespace= " ht tp : / /www . w3c . org/ svg " 
minOccurs=" 0 " 

processContents= 1 skip"/> 
<xs:any namespace= 

" ht tp : / / www • w3 . org /TR/ WD - rdf - synt ax# » 

minOccurs= 11 0 " processContents= ■ skip « / > 
<xs:any namespace= "http://www.iana.org/sdpng" 
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minOccurs«"0" 

processContents= 1 skip" /> 

<!-- 

further VoxML and other 
Presentation Markup Languages 
> 

</xs:all > 

<xs .-attribute name="id" type="xs : ID" use=" required" /> 
<xs : attribute name=:"type" 

type = " medi ait emAt t r ibut eTypeType " 

use=" required" /> 

<xs: attribute name="region" type="xs : IDREF" 
use = " required" / > 

<xs: attribute name^'src" type="xs:anyURI , j/> 
</xs : complexType> 
</xs : element > 
</xs : choice> 
</xs : complexType> 

<xs : simpleType name^"mediaitemAttributeTypeType" > 
<xs : restriction base="xs:string n ,> 

<xs : enumerat ion value = " audio " / > 

<xs : enumeration value=" video "/> 

<xs : enumeration value= " image " /> 

<xs : enumerat ion value= " text " / > 

<xs : enumeration value= 1 slideshow"/> 

<xs : enumeration value=" text ticker "/> 
</xs : restriction> 
</xs : simpleType> 
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<xs : element name=" interact ions n i " 

type= » interact ionsType " / > 

<xs:element name= "widget" type=» interact ionType"/> 

<xs : complexType name= "interact ionsType " > 
<xs: choice maxOccurs= "unbounded" > 
<xs : element name= "widget " > 
<xs : complexType > 
<xs : sequence > 

<xs: element name= "param" 
maxOc curs=" unbounded " > 

<xs : complexType> 

<xs : complexContent > 
<xs : restriction base="xs : anyType " > 
<xs: attribute name= n name" 
type= "xs : string " / > 

<xs : attribute name= "value " 
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type= "xs : anyURI " / > 

</xs : restrict ion> 
</xs : complexContent> 
</xs : complexType> 
</xs :element> 
</xs : sequence> 

<xs:attribute name= n id" type="xs:ID" 

* 

use= " required" / > / 

<xs:attribute name = "type" type="xs : string" 

use=" requi r ed " / > 
<xs: attribute name- "region" type="xs : IDREF" 

use=" required" /> 
<xs: attribute name= "class" type«"xs : string" 

use= " required" / > 
<xs : attribute name='src" type="xs -.anyURI" 
use- « required" / > 
</xs : complexType> 
< /xs : element > 
</xs:choice> 
</xs : complexType> 
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<par> 

<choose> 

<mediaitem idref ="..."/> 
<mediaitem idref 

</choose> 

<choose> 

<mediaitem idref ="..."/> 
<mediaitem idref ="_"/> 

</choose> 
</par> 
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{modelGroup=choice; 
maxQccursF=unbounded) 



«element» 

constraints 



id:ID 



— . ax 



{modelGroup=choice; 
maxOccurs=unbounded) 





«element» 






constraint 


— m 


id: ID 





«element» 

restriction 



value: xs: string 



«element» 

adaptationtrigger 



value: xs:string 



«element» 

adaptationfrequency 



value: xs: string 



\ 



toe 



908 



<xs: element name=" constraints" type- "constraint sType»/> 



<xs : complexType name* " const raint sType " > 
<xs: choice maxOccurs= n unbounded" > 
<xs : element name=" constraint " > 
<xs : complexType> 

<xs: choice maxOc,curs = "unbounded" > 
<xs: element name- " restriction" > 
<xs : complexType> 

<xs : comp 1 exCont ent > 

<xs:restriction base="xs :anyType"> 
<xs : attribute name =" value" 

type«"xs : string" /> 

< /xs : restrict ion>" 
</xs : complexContent> 
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</xs : complexType> 

</xs : element > 

<xs :element name^'adaptationtrigger 1 ^ 

<xs : c ompl exTyp e > 

<xs : complexCont ent > 



<xs: attribute name =" value n 
type= "xs : string "/ > 

</xs : restrict ion> 
</xs : complexCont ent > 
</xs : complexType> 
< /xs : e 1 ement > 



<xs rat tribute name= 11 value " 
t ype= " xs : s t ring 11 / > 

< / xs : res t r ict ion> 
< /xs : compl exCon t ent > 
</xs : compl exType> 
< / xs : element > 
</xs : choice > 
< /xs : compl exType > 
</xs:choice> 
</xs : compl exType> 



<xs: restriction base="xs :anyType"> 



<xs : element name= " adapt at ionfrequency " > » 
<xs : compl exType> 

<xs : complexCont ent > 

<xs : restriction base= n xs :anyType M > . 
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Adaptation Possibilities 


Property, e.g. Bandwidth 


showing ml and m3 


109,000 Bit/s 


showing ml and m4 


89,000 Bi't/s 


showing m2 and m3 


77,000 Bit/s 


showing m2 and m4 


23,000 Bit/s 
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Name of the association 


Relevant modules 


"Linking Mediai terns" 


Adaptation Module and 
Mediai tern Module' 


"Linking Mediai terns to 
Layout " 


Medialtem Module and Layout 
Module 


"Linkxng Widgets to 
Layout " 


Interaction Module and Layout 
Module 
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Name 


Description j 


value range 


units | 


applies 1 S 
to 


f ramerate 


restricts the I 
f ramerate to use for 
mediatiems of type 

"vi ceo I 


positive I 
integer 


frames/s J 


mediaitem j 


sampling - 
rate 


restricts the 
sampling rate for J 
mediatiems of type 
"audio" 


positive 
integer J 




mediaitem 


bitsper- 
pixel 


restricts the color 
depths for mediatiems 
of type "video" or 
"image" 


positive j 
integer 




mediaitem 


quality 


restricts the media 
quality to use. .This 
setting can be 
interpreted 
differently depending 
of the used codec for 
the media item 


float 
between 0 
and l 


/ 

/ 


mediaitem j 

1 


bitrate 


restricts the bitrate 
to use 


1 positive 
1 integer 


kBit/s 
MBit/s 


1 a11 


bitrate- 
toADDRESS 


restricts the bitrate 
for a specific target 
host. ADDRESS can be 
an IP address or a 
hostname . 


1 pUoXLlVC 

j integer 


kBit/s 
MBit/s 


all 


memory 


limits the amount of 
memory to use 


positive 
integer 


j Bits 
kBits 


all 



Fig. 11 



Name 


| Description 


value range 


units 


applies to 


cpu 


restricts the amounf 
of processing power 
to use. Either 

SDecif ied in th«a 1 naH 

percentage or an 
normalised processing 


positive 
integer 




all 


audio- 
quantity 


restricts the number 
of media items of 

^- j auuiu 


positive 
integer 




all except 
mediaitem 


video- 
quantity 


restricts the number 
of media items of 


positive 
integer 




all except 
mediaitem 


text- 
quantity 


restricts the number 
of media items of 


positive 
integer | 




all except 
mediaitem 


image - 
quant itiy 


restricts the number 
of media items of 
type image 


positive 
integer 




all except 
mediaitem 


discrete- 
quantitiy 


restricts the number 
of discrete media 
items 


positive 
integer 




all except 
mediaitem 


cont- 
quantity 


restricts the number 
of continuous media 
items 


positive 
integer 




all except 
mediaitem 
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Name 



f ramerate 



Description 



specifies framerate limits for 
media items of type "video" 



value 
range 



units 



positive 
integer 



frames/ 



bitrate 



specifies bitrate limits 



positive 
integer 



plr 



specifies limits for the packet 
loss rate 



positive 
float 



% 



delay 



specifies limits for the delay 



positive 
float 



ms 



jitter 



specifies limits for the jitter 



positive 
integer 
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<par> 

< choose priority="l"> 
<mediaitem idref=»Ml" 

pr i or i ty = n 3 11 / > 

<mediaitem idref= ,f M2" 

priori ty= " 4 " / > 
</choose> 
<choose> 

<mediaitem idref» M M3 n 

) 

priori t y= " 5 " / > 

<mediaitem idref =* ,, M4 n 

priori ty=* " 6 » /> 

</choose> 
</par> 

Fig. 13 



AP 1: M4, M2 

AP 2: M4, Ml 

AP 3: M3 # M2 

AP 4: M3, Ml 
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tsrco 

with priority 6-4 
with priority 6-3 
with priority 5-4 
with priority 5-3 
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<?xml version="1.0"?> 



Moo 



<amil> 
<head> 

^iroSt-layout width=«320" heigbLt="240" background -color= "white"/: 
<region irl "rfl" -np-"*" width^320» height-" 120 "/> 

«llion iL«rB"33t^ n top='-"rW^dth«"320" height="120"/> 

</layout> 

<mediaitems> ^ 

<mediaitera id="mA" type= "movie" reg*sqa=»rA" 

src= " rta» : //mediaserver /media .mov" / > 
</mediaitems> ^ 
<widgets> \ 



<widget id= "widget" 



^= WJ ^-^ r ype= "widget" region="rB" 
class= " de . cot icar . browser . widgets . SVGNavigation" 
src="http: l/\ ebserver/SVGNavigation. jar"> 

alue="http : //webserver/ view . svg" /> 



<paxam name="url" x 
</ widget > 
</widgets> 
</head> 

<body> I 
<adaptation> J 

<mediaitem idref«"mA"/> 
</adaptation> 
</body> 
</amil> . 
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yf?oo 



pirsentation-A-amfl 



v 


prcj«itatioo.B.amU^\_ 






^ Navigation Bar ^ 


\ 
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<xsl : t est cliff (current location, location ( '47,11'. , '9,34 •))< 
500 

<!-- do something -- > . . 
</xsl:test> 
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< event s xmlns="http: //www. sony.de/amil 11 

xml ns : xs 1 = n ht t p : / / www . w3 . org/ 1 9 9 9 /XSIi/ Format n 
xmlns:ev="http://www.w3 . org/2 OOl/xml -event a" > : 
<ev:listener event =" location: : changeEvent 11 > 
<action type="passToScript "> 
<xsl : choose> 
<xsl:when 

<xs l:test="diff( current 1 oca t ion / 1 oca t ion ( » 4 7 , 1 1 " , ■» 9 , 3 4.* ) ) " > 
<link> 

<src . id= "medial temA, regionA" > 
<target id=" medial temA, regionB"> / 

</link> 

<link> 

<element id= "mediai temB, regionB" > 
<region id= "mediai temB, regionC"> 
</link> 
</xsl :when> 
<xs 1 : o therwi se > 
<link> 

<src id=s"mediaitemA, regionA" > 
<target id=" medial temA, regionC"> 
</link> 
</xsl : otherwise> 
</xsl : choose > 
</action> 
</ev:listener> 
</ event s> 
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<events xmlns="http: //www. sony.de/amil " 

xmlns :xsl="http : //www. w3 . org/1999/XSL/Format ■ 
xmlns : ev= "http : //www . w3 . org/2 0 01/xml - 

events "> 

<ev : listener events" location: : changeEvent," > 
<action type="passToScript"> 
<link> 

<src id=" media itemA, regionA"> 
<target id=»mediaitetnA, regionC"> 
</link> 
</action> 
</ev: listener> 
</ event s> 



Fig. 2 0 



<par> 

<choose> 

<mediaitem idref =*"ml" > 
<mediaitem idref ="m2 " > 

</choose> 

<choose> 

<mediaitem idref ="m3 ,t > 
<mediaitem idref="m4 w > 

</ choose > 
</par> 
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<body> 

<constraints> 

<constraint id= f, low"> 

<restriction value* "bit rate > 100kbit"> 
<restriction value- "bit rate < ISOOkbit or 

framerate > 30" > 

restriction value="cpu < 40%"> 
< cons t raint > 
<constraint id* "mid" > 

<restriction value*" bit rate < 1000kbi,t n > 

restriction value="cpu < 30%" > 
< const raint > 
<constraint id*"high"> 

<restriction value*' 1 bit rate < 500kbit"> 

restriction value="cpu < 20%" > 
<constraint> 
<constraint id* "video" > 

<adaptationtrigger value=" framerate < 15 f/s"> 

<adaptationtrigger value* "plr > 0,05"> 
<constraint> 
</constraints> 
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<adaptation> 

<choose idref ="low"> 

<mi id= "medial temA" idref=" video" 

<mi id= » medial temB" / > 
</choose> 

<choose idref = "mid"> 

<mi id= "medial temC" / > 

<mi id= "medial temD"/> 
</choose> 

<choose idref = "high" > 
<mi id=r"mediaItemE f, /> 
<mi id= "medial temF"/> 
</choose> 
<adaptation> 

</body> 



Fig. 22 
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Calculation of all 'Adaptation Possibilities' 



filtering out adaptation possibnfflesX 
based on the 'current-situation' ) 



e.g. bandwidth consumption 



5 



Altering out adaptation possibilities^""* 
based on the priority J 



330g 



filtering out adaptation possibilities;^ 
based on the •current-situation*/ 



If more than one adaptation possibility is left 
on is selected at random. 




e.g. bandwidth consumption 



5 
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A MIL Document 
updated internal 
OnMng structure 



b)f 



HIorBj U ileal structural 



*cu»i mit-sJtuallmY has Chan god 
IWs results In triggering of on event 



w 



.current-situation' 



AMIL Document 



3 



.current-situation* 



^"j^AMIL Document^ 
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«element» 

mediaitem 


Linking Mediattems 
to Layout o..^ 


«e!ement» 

region 


region: IDREF 




id: ID i 
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«element» 

widgets 


Linking widgets 
to regions o..^ 


«element» 

region 


region: IDREF 




id: ID 
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«element» 

anchor source 


«link» 


«element» 

anchor destination 


attr: BDREF 


attr : ID 
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.«element>:> 

adaptation 



{modelGroup=choice} 



id:ID 



{modelGroup=choice, 
maxOccurs=unbounded} 



«element» 

mediaitem 



idrefclDREF 
priority:[1..99] 
startmode: startmodeType 
starttime:\d{l)+s 



«elemem» 

controlelement 



«clement» 

par 



priority:[1..99] 



«element» 

seq 



priority:[ 1..99] 
<S • 



«eleraent» 

choose 



priority :[1 ..99] 
startmode: startmodeType 
starttime:\d{l}+s 
onremove: onremoveType 
looping: xsiboolean 
empty: xs:boolean 
evaluation: evaluationType 



3908 



■ 1 

<<enumeration>> 

startmodeType 


«enumeration» 1 

onremoveType 


value » { restart, resume, laststop, 
playtime, contplay time } 


value 853 { destroy, remove } 



«enumeration» 

evaluationType 



value = { startup, con,\d{l}s > 
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<xs; element name* 11 adaptation" type* "adaptait ionType"/ 

<xs : complexType narae= » adapt at ionType 11 > 

<xs:choice minOccurs="0" maxOccurs*" unbounded" > 
<xs:element name*" choose" type*"chooseType" /> 
<xs:element name*"par" type* 11 parType " /> 
<xs : element name= 1 seq" type* 1 seqType" /> 
<xs .-element name*"mediaitem" type="mediaitemType 
</xs:choice> 

<xs rattribute name* "id" type*"xs :ID"/> 
</xs ; complexType> 

<xs: complexType name= " chooseType " > 

<xs : choice minOccurs^ " 0 " maxOccurs^ "unbounded" > 
<xs: element name=» choose" type* " chooseType "./> 
<xs: element name= ,, par" type= " parType " '/> 
<xs : element name= 1 seq" type= 1 seqType " /> 
<xs: element name*"mediaitem" type*"mediaitemType 
</xs:choice> 

<xs : attribute name*" priority" type*" priori tyType" / 
<xs : attribute name* ■ s tar tmode " type* • s tartmodeType 
<xs rattribute name*" start time" type* • start timeType 
<xs : attribute name*"onremove" type="onremoveType" / 
<xs : attribute name* "looping" type*"loopingType» /> 
<xs rattribute name*" evaluation" 
type* » evaluat ionType "/ > 

<xs rattribute name= 11 empty " type*"emptyType"/> 
</xs : complexType > 
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<xs: complexType name="paj:TypeJ , > 

<xs : choice minOccurs= n 0 ■ maxOccurs= "unbounded" > 
<xs:element name=" choose" type="chooseType"/> 
<xs:element name=«par" type="parType" /> 
<xs: element name=»mediaitem" type="mediaitemType" /> 
</xs :choice> 

<xs: attribute name= "priority" type= "prior ityType"/> 
</xs : complexType> 

<xs : complexType name= ■ seqType" > 

<xs : choice minOccurs= " 0 " maxOccurs= "unbounded" > 
<xs:element name="choose" type="chooseType"/> 
<xs: element name="par" type="parType"/> 
<xs : element hdrae= 1 seq" type= 1 seqType " / > 
<xs:element name=" medial tern" type="mediaitemType»/> 
</xs:choice> 

<xs:attribute name- "priority" type="priorityType»/> 
</xs : complexType > 

<xs : complexType name="mediaitemType"> 
<xs : comp 1 exCont ent > 

<xs : restrict ion base= " xs : anyType " > 
<xs: attribute name=«idref " type="xs : IDREF" 

use- " required" / > 

<xs:attribute name- "priority" type- "prior ityType»/> 
<xs:attribute name- 1 starttime" 
type- 1 starttimeType"/> 
</xs : restriction 
</xs : complexContent> 
</xs : complexType > 



Fig- 29 



<xs : simpleType name= ' startntodeTyp.e" > 
<xs; restriction base*"xs :string"> 
<xs: enumeration valuer "restart" /> 
<xs: enumeration value* "resume "/> 
<xs : enumeration value* " 1 as t stop n / > 
<xs : enumeration value* "playtime " /> 
<xs .-enumeration valuer "contplay time" /> 
</xs : restriction 
</xs : simpleType > 



<xs : simpleType name* » starttimeType"> 

<xs: pattern value* "\d{l}+s"/> 
</xs : simpleType> 

<xs : simpleType name= " onremoveType " > 
<xs : restriction base="xs :s.tring"> 
<xs : enumeration value* "destroy" /> 
<xs : enumeration value= 11 remove " / > 
</xs : restriction 
</xs ;simpleType> 

<xs: simpleType name= " loopingType M > 

<xs: restriction base*"xs :boolean«/> 
</xs : simpleType> 



<xs : simpleType name = » evalua t ionType n > 
<xs: restriction base*»xs : string" > 
<xs : enumeration value* ' start-up" /> 
<xs .-enumeration value* "con" /> 
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<i and integer values concluded with the unit 

< /xs : restriction 
</xs : simpleType> 

<xs : simpleType name= " emptyType " > 

<xs -.restriction base="xs : boolean" /> 
</xs : simpleType> 

<xs : simpleType name= "priori tyType ■ > 
<xs : restriction base= "xs : integer" > 
<xs : minlnclusive value= " 1 " / > 
<xs:maxlnclusive value="99"/> 
</xs : restriction> 
</xs:simpleType> 
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media item 

A(hq) 4 highquality high quality , 

\ /t 

A ( mq j. . middle quali ty middle qualit y 

decrease "N^ /increase 
A(lq)+ rfb ^*^»» X low quality /of ban**,* 
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media torn 
A 

b" "- ■ 
c 



Aja_ 



BJU_ 
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media item 
A | *fl) Af t) 

B — 
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Docuaen t Pa c t or 



creation £^ 
oz the AMZX. 
DOM, baaed 
an an am it 



cxeateDocument O » Document 



createBlement (Vocabulary) ■ Element. 



appendChild(Node) mode 



cobbbIc the modification 
executed on the AM XI* DOM 
to che supporting middle* 



commit () ivoil^ 



destroyDocuroent (Document! 



1 



Document 



"0 



XI. 

I 
I 
I 
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renoveChildWode) *Hode 



i 

createBlement (Vocabula ry) » Element^ 1 

■ T3 



appendChild (Mocae ) « Bode 



commit (Node) i void 



~0 

I 

~0 



Fig. 3 6 



37co 



1 II 1 








1 


1 1 


1 



Uudfor » specific .cuiraiWita-iloi 



Used fori 
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«event» 
'current-situation' changed 
(abstract) 



3&XI 



«eVent» 
noAdaptation Possibility 




«event» 
velocityChanged 



3soe 



«event» 
locationChanged 
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1~* 1 {modelGr oup=choice) 




{ raodeIGroup=choice} 

- 33oa 



The name cf o typa has to ccntorm to the pattern: 
•«finso/ z Ynreo of fl» event 



5 



<<«nuraenrthm>> 

EventTyp e 



X{modelOro up=choice) 



conditional 



{modeIGrou p=seguertce) 



source 



id; IDREFS 



target 



id IDREFS 



<xs : element name= » adapt at ionevent 11 ~ r 

type= » adapt at ionEvent Type »/ > 

<xs : complexType name= " adapt at ionEvent Type" > 
<xs : choice minOccurs="0"> 

<xs : element name^'listener" type:="listenerType n /> 
</xs:choice> 

<xs:attribute name=*"id" type= n xs:ID" /> 
</xs : complexType > 

<xs: complexType name= » listenerType" > 
<xs:choice> 

<xs:element name=" action" type="actionType»/> 
</xs:choice> 
</xs : complexType > 



Fig. 39 
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<xs: complexType name^'actionType" > 

<xs:choice> 

<xs:element name^'link"' type="linkType ,f /> 

</xs : choice> 
</xs : complexType> 

<xs : complexType name= " 1 inkType " > 
<xs : sequence> 

<xs : element name= ' source " type= 1 sourceType " / > 
<xs: element name=" target " type= H targetType"/> 
</xs : sequence> ' 
</xs : complexType > . 

<xs : complexType name= ■ sourceType M > 
<xsd : complexCont ent > 
<xsd : restriction base= "xsd : anyType " > 

<xsd : attribute name= n refs" type="xsd: string" /> 
</xsd : restriction> 
</xsd : complexContent > 
<xs : complexType > 

<xs : complexType name= " t argetType " > 
<xsd : complexContent > 
<xsd : restriction base= "xsd : anyType " > 

<xsd: attribute name="refs" type="xsd: string" /> 
</xsd : restriction> 
</xsd : complexContent> 
<xs : complexType >. 
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<xs:simpleType name= " event Type " > 
<xs: restriction base="xs : string"> 

<xs : enume rat i on 
value= "adaptation: :noAdaptationPossiblity" /> 

<xs : enumeration value= 0 application : : newMediaitem" / 

<xs : enumeration 
value= » location : : velocityChangeBvent " / > 

<xs : enumera t ion 
valuer "location: : locationChangeEvent " /> 

</xs : restrict ion> 
</xs t simpleType> 



Fig. 39 
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<?xml version="l . 0"?> 
<amil :amil 

xmlns : rdf ="http : / /www . w3 . org/TR/WD-rdf -syntax#" 
xmlns : ami 1 = " ht tp : / /www . sony . de/ amil " 

xmlns : xsi= " ht tp : / /www . w3 . org/ 2 0 0 1/XMLSchema- instance ■ 
xmlns : smi 1 = " ht tp : / / www . w3 . org/ 2001/ SMIL2 0 /WD/ Layout 
Module" 

xmlns : svg= "http : //www . w3c . org/ svg" 

xmlns :xhtml="http: //www. w3 .org/1999/xhtml"> - - 
<amil:head module> 

<amil : title>Example</amil :title> 

<ami 1 : 1 ayout > 

<smil : root-layout width="640" height="480" background- 
col or = whit e " /> 

<smil:region id= "upperpart " left="0" top="0" width="640» 
height-" 240 "/> 

<smil: region id="middlepart " left= n 0" top="240" 

width="640". height="200"/> 

<smil: region id="lowerpart " left="100" top="440" 
width="440" height="40"/> 
</amil : layout> 

<amil : medial terns > 

<amil :mediaitem id="mediaA-hq " type=" movie" 
region= " upperpart " 

src= " rt sp : / /mediaserver/centa/AKnightsTale -hq . mov" /> 

<amil:mediaitem id="mediaA-mq " type="movie" 
r egion= " upperpart " 
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src= ,, rtsp://mediaserver/centa/AKnightsTale-mq.mov"/> 

<amil:mediaitem id= n mediaA-lq " type= "movie" 
region= "upperpart " 

src= " rt sp : //mediaserver/centa/AKnight sTale- Iq . mov" /> 

<amil:mediaitem id="ad" type= " text /xhtml " 
region= "mi ddl apart " > 
<xhtml> 

-.plain text in XHTML _ 
< /xhtml > 

</amil :mediaitem> ..... 
< /ami 1 : medial terns > 

<interactions> 

<widget id=" widget A" type= n class M region= n lowerpart» 

class= n de . comcar . browser . widget . SmartWidget n / > 
</interactions> 
</amil:head module> 

<amil:body module> 
<adaptation> 

<amil:mediaitem idref =" ad"/ > 
<amil:widget idref =" widget A" /> 
<choose> 

<amil :mediaitem idref ="mediaA-hq"/> 
<amil rmediaitem idref ="mediaA-mq ,t /> 
<amil :mediaitem idref ="mediaA-lq"/> 
</choose> 
</adaptation> 
</amil:body module> 
</amil:amil> 
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